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MEMBERS OF THE EVALUATION SECTION OF HARVARD PROJECT 
PHYSICS. DESCRIBING WHAT IS SAID TO BE THE FIRST ATTEMPT TO 
select a national random sample of (HIGH SCHOOL PHYSICS) 
TEACHERS. LIST THE STEPS AS (1) PURCHASE OF A LIST OF PHYSICS 
TEACHERS FROM THE NATIONAL SCIENCE TEACHERS ASSOCIATION (MOST 
complete AVAILABLE) . (2) SELECTION OF 136 NAMES BY A TABLE OF 
RANDOM NUMBERS DERIVED FROM ORDINAL NUMBERING OF THE 1S.7D2 
LISTINGS FOR THE CONTINENTAL U.S.. (3) SENDING OF A 
DESCRIPTIVE INVITATION TO PARTICIPATE IN EXPERIMENTAL 
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1966 NSTA LIST WHO DID NOT MAKE SUMMER COMMITMENTS MORE THAN 
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GROUP ARE ALSO DISCUSSED. (AF) 
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a Jntroduc tion 

s 

g Given the task of evaluating the effects of a new high 

^chool physics course, the evaluation group of Harvard Pro- 
fit 

ject Physics* attempted to (a) select the participay^g 
physics teachers from the national population on a truly 
random basis and (b) to assign them rcuidcxnly to experimental 
and control groups. The random selection was particularly 
appropriate for the Project Physics course because it is 
intended for a very diversified audience. Randomness jus- 
tifies statistical inferences to a much larger population 
than is reasonable for a group of volunteer teachers. 



I In this paper we will describe what is to our know- 

j ledge the first attempt to select a national random sample 

of teachers. The discussion of the problems encountered 
and the success and possible value of the undertaking may be 
helpful to other curriculum research and evaluation groups. 




A description of the materials and philosophy of Harvard 
Project Physics, see The Physics Teacher". 5. 5 (Mav 19671. 
pp. 197-233. “ 



- 2 - 



A list of the names and addresses of 16,911 physics 
teachers was purchased at a cost of $255 from the National 
Science Teachers Association (NSTA) , a group which maintains 
the U# S. Registry of Junior and Senior High School Science 
and Mathematics Teaching Personnel. The NSTA has reported 
(1965) that the list is compiled from responses received 
from 81 percent of all secondary schools in the United States. 
We were unable to find a more complete, or even comparable, 
list from any other source. 

Because of travel costs for teacher training, we lim- 
ited our population to the 16,702 physics teachers listed 
for the continental United States. Numbers were assigned 
to each of the teachers according to their ordinal position 
on the list and a table of randOTi numbers was used to select 
a total of 136 nanes. These two steps took a clerical 
assisteuit approximately four days to complete. (Actually 
the selections were made in two blocks. Initially 85 names 
were selected and the teachers contacted. Then to complete 
the roster of teachers available for summer training, a 
second random selection of 51 names was made.) 

A registered letter describing the curriculum project 
was sent to the teachers together with an invitation to 
participate in an experimental evaluation of the course. 
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They were informed that those teachers agreeing to partici- 
pate would be randomly assigned to one of two groups— an 
experimental group and a control group. Thus the teacher 
would have to be willing to be in either group if he con- 
sented to participate at all. 

A description of the responsibilities of both groups 
was given in the letter. In summary, the experimental 
group would attend a six -week training session, take a 
series of tests, teach the course during the academic year 
1967-68, and administer pre, mid, and post tests to their 
physics students. The control gr:>up would attend a two-day 
briefing session, take a series of tests, and administer 
the same battery of tests to their students? but they would 
continue to teach their regular physics courses. Travel 
expenses, summer school stipends, and course materials were 
to be provided by the curriculum project. 

— 

For reasons of expense and on the basis of a research 
design that was tried out with volunteer teachers during 
1966-67, we decided to assign 40-45 teachers to the experi- 
mental group 2 uid 20-25 teachers to the control group. Thus 
it was planned that the evaluation would include 60-70 
teachers with an estimated 3,000 to 4,000 students. 
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The teachers contacted were requested to return a one- 
page questionnaire r indicating either their willingness to 
participate in the experiment or their reasons for refusal. 

To take advantage of the opportunity to query a random sample 
of physics teachers, the questionnaire included several ques- 
tions about school size, teaching load, physics enrollment, 
and text used. Several of these findings are reported by 
Welch (1967). 

The invitation letters were sent by registered mail 
and replies were requested within two weeks. Uncertain as 
to what response rate to expect, we nailed 85 invitations. 
Responses were received from 68 within two T/eeks. Six 
registered letters were returned unopened with various post 
office indications: ''addressee unknown," "unclaimed,” or 
"moved — left no address." The remaining 17 were telephoned. 
Of the> 2 j, six said yes, nine said no, and we were unable 
to locate two because of incorrect addresses. Thus in our 
first mailing, 77 teachers were contacted; 46 agreed to 
participate; 30 had prior commitments which prevented their 
acceptance; and 3 were not interested. Because the response 
rate of the first mailing was about 60%, a second mailing 
of 51 invitations was made. We hoped this would give us 
approximately 70 teachers for the study. 
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A similar pattern of response, telephone calls, accept- 
ance and rejection followed the second mailing. It is con- 
venient here to talk about the group as a whole, although 
nearly five weeks elapsed between the first mailing (January 
3) and the last acher reply (February 9) • 

A total of 136 letters of invitation were mailed, but 
only 124 teachers actually were contacted. Nine letters 
were "returned to writer" by the post office and three others 
could not be reached by phone. These twelve names were 
omitted from the sample. Replies were received from 87 of 
the 124 teachers within two weeks after mailing. The re- 
maining 37 were contacted by telephone to determine: (l)lf 
they had received our letters, and if so (2) why they had 
not answered. Reasons for not responding ranged from "T'That 
letter?" to "I thought it was a joke I" to "I didn't bother 
to read the letterl" Eventually all 124 teachers replied 
to the invitation. 

To us, the response was most gratifying. Only six 
teachers refused because they were not interested. Seventy- 
two agreed to participate according to the conditions speci- 
fied, while 46 were unable to accept because of prior com- 
mitments. The nature and frequency of these prior commit- 
ments are listed below: 

ERIC 
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A. Continuation of work on Master's degree . 11 

in summer school 

B. Mo longer teaching physics 11 

C. Stammer job commitment 10 

D. Physics not offered in 1967- o at their 5 

school 

E. Health reasons prevent extended travel 3 

P- Misc. (changing jobs, expecting baby, 5 

etc.) 

46 



In a number of instances when the teacher we attempted 
to contact was no longer teaching, a substitute teacher was 
offered by the school administration. However, we realized 
that accepting substitutes would not be consistent with 
randomization. The sine qua non of random selection is 
equal selection probability for all subjects. Teachers in 
the school system where a selected teacher no longer taught 
physics would have a higher selection probability than other 
teachers: a certain probability because of their own name 
on the master list and a certain additional probability 
because of being in the same school system as the selected 
but unavailable teacher. If substitutions were accepted, 
there would be a selection bias in favor of school systems 
with high turnover. 




We are concerned about bias that might be introduced 
in the sample, so several characteristics of the acceptors 
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were compared with those who were unable to accept. A two 
tailed t-test was used to test the statistical significance 
of differences. These comparisons are summarized in Table 1. 

[Insert Table 1 about here.] 

Questionnaires were returned during 1966-67 from 117 
teachers— 71 acceptors and 46 non-acceptors. Apparently, 
teachers who accepted the invitation, when compared to the 
non-acceptors,- are more likely to teach in larger schools 
and to be currently teaching the Physical Science Study 
Committee (PSSC) physics course. It seems reasonable to 
Interpret these differences as a greater receptiveness to 
innovation itn larger schools where previous innovations have 
been accepted. 

A table of random numbers was used to assign 46 of the 
teachers to the experimental group and 26 to the control 
group. Letters describing the six-weeks summer session at 
Wellesley College were mailed to the experimental group 
together with letters of agreement to be signed by school 
officials. The control group teachers were invited to attend 
a two-day briefing session in Cambridge at which time their 
role in the evaluation was described. 

2 

PSSC is a recently-developed physics course— one of the 
first of the national curriculum projects. 
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In the four months that followed the assignment to 
experimental and control groups, attrition occurred in 
both groups. Among the 46 teachers of the experimental 
group, one died suddenly, three quit teaching, one went to 
teach in Uganda, two were transferred to new positions, two 
could not come to terms j/ith their school boards, and one 
became too ill to travel. Thus the experimental group was 
reduced to 36, or 78% of the initial group. We decided that 
this number was still adequate for our evaluation, and to 
avoid introducing bias we did not supplement the group in 
any of several ways that were considered. Five teachers in 
the control group dropped out for reasons similar to those 
above, leaving a total of 21 teachers (81%). Again this 
number was considered adequate for the research planned. 



In 1966 similar attrition among volunteers occurred 
prior to a Project Physics sunimer briefing session. In that 
case, a sample of 45 teachers v;as selected from among vol- 
unteers to attend a summer session and then teach Project 
Physics. However, only 37, or 82%, actually attended the 
institute. Thus we are inclined to believe that the attri- 
tion of 20% over a few months is not peculiar to national 
random samples. 



Our final sample consisted of 57 physics teachers. 
Thirty-six of these attended the summer training session 



-9- 



and are currently teaching the course. Another group of 21 
attended the two-day briefing session and are presently 
teaching their regular physics courses. Data were gathered 
using a system of randomized data collection within each 
cl^ss which maximizes the number of tests and minimizes 
testing time for individual students (Walberg and T'^Telch, 
1967) . Data are n^ w being collected on the 3000 students 
being taught by the final sample of 57 teachers. 



Discussion 



To claim a random sample, the population over which 
the sample is random must be specified. Our population is 
not, to begin with, "all high school physics teachers" 
bat zather "al”’ high school physics teachers on the 1966 
NSTA list." There is undoubtedly some bias introduced by 
the estimated 20% of high school teachers who were not in- 
cluded in the list. Another bias is introduced by the ex- 
clusion of the 37% of the initial sample vrhich had previous 

3 

In addition to the 57 new teachers selected at random, 21 
teachers experienced in teaching Project Physics are in- 
volved in the research design. A total of 4,067 students 
are being tested on approximately 80 different variables. 



- 10 - 



conraitments • Since the prior commitment was often a sunnier 
institute we might infer that our final sample was biased 
toward the less €Ut>le teacher. There were a substantial 
number of withdrawals after the assignment to groups. 

Although there is no clear distinction between the experi- 
mental and control, we must infer that the overall sample 
is again biased. We could gia racterize t h e populatio n, then, 
as all high school physics teachers on the 1966 NSTA list 
who did not make summer commitments more than 3 months in 
advance an'^ who would not back out of the research agreement. 
Certainly this population is more limited than would be ideal, 
but it is the best that we could possibly get and as far 
as w€ know it is better them any that has been used before. 

Added to the possible biasing factors is the extra cost 
involved in randoTv sampling. Purchasing the list of teachers, 
extra mailing costs, telephone calls, selecting the names at 
random, and mamy other clerical problems increased the cost 
approximately $1000 above what it would cost to use volun- 
teers. The cost of bringing the 21 control group teachers 
to Cambridge for two days was $3500, an expense that could 
have been avoided if volunteer teachers were used. Further, 
travel costs of $5000 were required to bring the experimental 
group to the six-weeks session. This is approximately $3000 
more than v?ould be required for a regional sample. Thus the 
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total increase in cost by using a national r^dcxn sample 
- an estimated 

aSOOO. 



In general/ we believe that our national san^ling was 
successful. The teachers are cooperative and willing to 
assist in every way we have asked. The experience of work- 
ing with them during the summer was rewarding and informa- 
tive to the physicists and educators developing the course. 
We have been able to obtain information from a national 
sasqple of physics teachers. With this data we are able to 
make some generalizations to a national population on 
educational experience/ training/ teaching situation/ know- 
ledge of physics / and attitude towards teaching. AIjO/ 
poss ible differences between these teach^^rs amd the vol- 
iwteer teachers selected the precceding year (Walberg and 
Welch/ 1967) can be identified. These data should be help- 
ful in^udging the valu es and disadvantages of using volun- 
teers for course development^ a question which bears upon 
virtually all past evaluation of curricula designed for 
national usage. To the extent that the selected teachers 
have a representative sample of students/ it will be 
possible to draw inferences concerning the attitudes/ 
interests/ euid ability of students enrolled in high school 
physics. Host important/ we are now able to try a new course 
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project in a variety of schools that cire representative of 
all secondary schools. 

An Unexpected result has been the desire of those in- 
volved in writing the course to want to try out new ideas 
and materials on students of the randomly selected teachers 
rather than on the students of volunteer teachers. However# 
we have resisted the urge to try new things on these classes 
for fear of disturbing the experiment. Aside from the 
statistical reasons for taking national randcm samples# it 
would seem advisable to use such samples for subjective 
evaluation of course materials in their early stages of 
development . 

We cannot say with certainty at this time that the 
invited teachers (experimental and control) will be consis- 
tent in their cooperation throughout the year. However# 
our experience with them thus far has given us little cause 
to worry. We hope to report at the conclusion of the 
evaluation that the random sample was worth the expense and 




effort. 




Table 1 







Comparisons Between Acceptors and Non-Acceptors of the 
Experimental Evaluation of Project Physics 



Non- 



Variable 


Acceptor 

Mean 


SD 


Acceptor 

Mean 


SD 


Differ- 

ence 


t value 


Grade 10-12 
Enrollment 


786 


753 


442 


449 


334 


2.77* 


Number of 

Physics 

Sections 


2.3 


1.7 


2.0 


3.5 


.31 


.65 


Physics 

Enrollment 


53 


54 


39 


86 


14 


1.03 


Fraction of 
12th Graders 
Taking 
Physics 


.20 


.25 


.26 


.24 


.02 


.33 


Fraction of 
Phys** cs 
Students 
TcJcing PSSC 
Physics 


.43 


.75 


.16 


.56 


.27 


2.09* 


Fraction of 


.15 


.15 


.20 


.57 


-.05 


-.70 



Girls in 
Physics 



*p<.05 



^The Non-Acceptor group includes the 46 teachers unable to 
participate because of prior cOTiroitment and the 6 who refused 
due to lack of interest. 
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